5. AUTOMATIC CHECKOUT AND CONTROL 
Joseph N. Sivo 

As explained in the paper by S. C . Himmel, careful testing and checkout of all 
vehicle systems are necessary to achieve success in the launch vehicle business. 

This testing procedure starts during the manufacture of parts and systems and it 
continues on the complete vehicle up to the moment of launch. Complete systems 
testing has contributed greatly to the success of the NASA space program. Although 
such a thorough checkout requires large expenditures of manpower and money, the 
end result - successful launches - justifies the approach. 

Overall launch operations can be divided into three phases: prelaunch checkout, 
launch countdown or vehicle startup, and postlaunch control or steady-state opera- 
tion. Experience has shown that the prelaunch phase is the most critical of the three 
and creates the greatest demand on manpower and equipment. If proper prelaunch 
checkout is provided, the startup phase, by comparison, becomes somewhat routine. 
This can, in most cases, also be said of the postlaunch or run phase. Therefore, 
most of the discussion herein is concerned with this prelaunch phase. 

In the early days, rockets were relatively small single-stage vehicles. The 
Atlas booster (fig. 5-1) was used in the manned Mercury program and is somewhat 
typical of the smaller boosters. Checkout of all the vehicle systems was a manual 
operation, and launch crews were relatively small. 

The Saturn V (fig. 5-2) is the three-stage vehicle soon to be used in the manned 
Apollo Moon program. It has eleven main engines - five in the first and second 
stages and one in the third stage - as compared to three main engines on the Atlas. 
The Saturn is quite large - standing some 350 feet high as compared to about 100 feet 
for the Atlas. The combination of multiple stages, their attendant interfaces, the 
need for in-flight stage separation, and the vehicle size combine to increase the 
number of system parameters to be checked a hundredfold over the Atlas. 

A size comparison between the Mercury capsule and the Apollo spacecraft also 
serves to show the increased prelaunch checkout that is necessary. The Mercury 
capsule was a one-man single-stage device weighing about 3000 pounds used to orbit 
the Earth. The Apollo spacecraft, in contrast, is a four-stage system - a capsule 
to house three men on the trip from the Earth to the Moon and back, a stage to retro 
into and depart from a lunar orbit, and a two-stage two-man vehicle to land on and 
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launch from the Moon’s surface. When fueled, the spacecraft weighs about 

100 000 pounds. On the Mercury capsule, 88 system parameters were checked prior 

to launch. This number has increased to nearly 2600 checks on the Apollo system. 

The change in vehicle and spacecraft size with the Apollo program reflected it- 
self in the launch complex necessary to support them. Figure 5-3 shows the typical 
Atlas launch complex used for such programs as Mercury. The vehicle was assem- 
bled on the pad and checked out manually, and the launch was directed from a block- 
house only a few hundred yards away. 

The Saturn complex shown in figure 5-4 is a different matter. The vehicle and 
spacecraft are assembled in the Vehicle Assembly Building, one of the largest build- 
ings in the world, and they are then transported fully assembled to one of two launch 
pads - one is 3 miles distant, and the other is 5 miles. This entire launch complex 
covers over 30 square miles. 

Experience had shown that the checkout of all systems prior to launch was man- 
datory if high reliability was to be achieved, but the vehicles have grown so in com- 
plexity and size that the proven manual checkout techniques were unable to cope with 
the increase in elements to be checked. The only recourse available was automation 
of checkout procedures wherever applicable. Automation in itself was no cure all, 
and it carried with it many new problems. However, the problems created by auto- 
matic or computer- controlled test and checkout are being resolved, and the ability 
to check out vehicles as complex as the Saturn V and its spacecraft has been demon- 
strated. 

To give an idea of the complexity involved, table 5-1 presents the number and 
type of measurements taken on the Saturn launch vehicle during checkout. Values of 
acceleration, vibration, temperature, pressure, etc. are taken throughout the test- 
ing and total some 2600 measurements including both analog and discrete. 

The need for automation to conduct the tests that help to ensure reliability has 
been discussed, but questions such as the following must yet be answered: What is 
meant by the automatic checkout system? What are its major components? How 
does it work? What types of tests can be run? 

Figure 5-5 shows a typical automatic checkout system that is currently in opera- 
tion for the checkout of the Apollo spacecraft. This system is referred to as the 
Automated Checkout Equipment or ACE system and contains two main functional 
parts - a Command System and a Display System. 

Test commands emanating from the test consoles are received and processed by 
the command computer which, in turn, sends appropriate commands through the 
transmission equipment to the vehicle under test. 

The output of the instrumentation on the vehicle and the responses to the test 
stimuli are transmitted to the data receiving and recording equipment of the Display 
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System. Some of the data are routed through a data processing computer where they 
are made ready for display in engineering units on the test console cathode ray tube, 
sometimes referred to as a CRT. The data are also routed directly to the support 
consoles for presentation on conventional display devices. 

A few examples of how to implement the testing with such a system help to ex- 
plain how the system is utilized. Currently in the power industry, one of the proce- 
dures associated with startup of a turbogenerator unit is the gradual heating of cer- 
tain parts of the turbine with steam to minimize thermal shock. This usually in- 
volves cracking the steam supply valve to allow steam to enter the turbine slowly. A 
somewhat similar procedure is used when cryogenic propellants are loaded. How- 
ever, very low temperature thermal shock is the problem here. 

Figure 5-7 shows a schematic of the hydrogen loading system of the Saturn V 
vehicle. The loading process is initiated by an engineer at the console pressing a 
start button. A vacuum-jacketed storage tank supplies hydrogen to the vehicle 
through either the chilldown valve, which controls the initial fill rate, or through the 
main fill valve to another pair of valves near the vehicle. Propellant normally flows 
through the fill valve into the tank. As the tank nears capacity, flow is metered 
through only the replenish valve at a low rate to complete the filling process. Pro- 
pellant that is lost due to boiling in either the vehicle tanks or in the main storage 
tank is piped to a burn pond for disposal. The lost propellant in the vehicle tanks is 
replaced through the replenish valve. All the valves are operated electropneumati- 
cally. Level sensing probes of the continuous reading type are installed in each tank. 

Previous calibration tests on similar vehicle tanks provided information on max- 
imum allowable fill rates as a function of tank liquid level that avoid excessive ther- 
mal stresses due to cold shock. With these data, a propellant loading computer pro- 
gram was generated that applied all the limitations imposed on the filling process 
determined in the calibration tests. The program is a precisely organized sequence 
of events triggered by specific tank liquid levels. 

In the actual tanking operation, outputs from the level sensors are sent to the 
computer and compared continuously with the reference standard stored in the com- 
puter memory. During the loading process, .fill rates are changed depending on the 
liquid levels. For example, in the second-stage hydrogen tank, a small amount of 
hydrogen is introduced into the tank to cool the tank slowly to -160° F, and then a low 
initial fill rate of 1000 gallons per minute is used up to the 5-percent level. Next, 
the rate increases to 10 000 gallons per minute until the 96 -percent level is reached; 
then, it goes back to 1000 gallons per minute to the 99-percent level; and finally, it 
is topped at 500 gallons per minute. If a malfunction occurs, the filling process 
stops and manual valve control is possible from the control console. Propellant 
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loading is a precise and somewhat delicate process that has been found well suited to 
automated control. 

On a vehicle like Saturn V, there are hundreds of specific events which must be 
checked as a function of time to assure that things are progressing in an orderly pre- 
determined fashion. One of the specific event readout panels for the first stage is 
shown in figure 5-8, which is not meant to be read but only to illustrate how events 
are displayed in the control room. As each event occurs, for instance an oxygen 
purge valve opens, an appropriate light comes on and its companion light, the oxygen 
purge valve closed, goes off. When one considers that this purge valve may open and 
close several times during normal operation of the oxygen system, it is necessary to 
know not only whether the valve is open or closed, but also what should it be at the 
given point in time. This particular panel monitors 250 separate events and is only 
one of several like it. Although it does provide a real time presentation of all events 
as they occur, it is quite clear that an operator or many operators would be unable to 
visually keep tabs on all the events as they occur in real time. 

The computer checkout system for the Saturn V has the capacity of scanning 
4300 discrete events such as these every 2 milliseconds. As the scan progresses, 
the condition of each discrete signal is compared by the computer to the status it 
should have at the time of scan. If there is a disagreement in status, the test con- 
ductor is immediately notified of the anomaly. Depending on the anomaly, appropri- 
ate action is taken automatically by the computer if previously instructed to do so. 

Not only can the computer keep track of discrete signals as they occur, but it can 
conduct a test program that will exercise each discrete event to check if all are func- 
tioning properly. 

Experience has shown that human error has a habit of being a rather unpredict- 
able source of system failure. Although testing all parts of a complete system is of 
great value, the more testing that is done, the greater the chance of a human error 
causing failure. 

The following example illustrates the type of error that can and does result from 
not following instructions exactly. Figure 5-9 shows an Atlas flight programmer that 
controls the sequence of events occurring on the booster after liftoff. The test that 
is run routinely on the programmer requires that both ac and dc power be connected 
to their respective input terminals at the start of the test. 

Portions of the actual checkout sheet used while conducting the test are given in 
figure 5-10. The sheet specifies the order that the power leads are to be energized. 
The dc power must be energized before the ac power if load limits on components 
within the programmer are not to be exceeded. In other words, if the ac power is 
connected first, you overload the programmer. Certainly, sufficient instructions 
have been given to caution the checkout crews. Unfortunately, the programmer was 
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overloaded anyway. The cause was human error. Had this particular test been per- 
formed under automatic checkout, the failure would have been avoided. This is a 
typical case of a routine test that had been run a hundred times without any trouble 
suddenly causing a loss of both time and money. Once a routine test has been 
checked out using the computer and the computer test program verified, the test is 
always conducted in the same way no matter how many times it is run. 

Another major problem in checking out complex systems becomes one of data 
handling. The more instruments that are used and the more tests that are run, the 
more that is learned about the systems. However, as usual, nothing is free. Instru- 
ments and tests generate data that must be analyzed if value is to be derived from the 
effort. This can become an overwhelming task using conventional techniques. How- 
ever, having a computer available as part of the checkout system allows online data 
processing at extremely high rates. 

For example, one of the critical decisions that must be made prior to launch is 
whether the winds aloft will cause structural damage to the vehicle due to vehicle 
bending. The question is handled in a very direct manner. Some time prior to 
launch, a wind sounding balloon, similar to the one shown in figure 5-11, is released 
at Cape Kennedy. Its motion is tracked by radar as it rises. Data from the radar 
are recorded on magnetic tape, which is fed to the computer. The computer con- 
verts the radar data to wind velocity and direction and then plots them as functions of 
altitude as shown in figure 5-12. The particular wind shown here reached a maxi- 
mum speed of 250 feet per second or about 170 miles per hour at an altitude of 
38 000 feet. The wind direction was nominally offshore. 

The effect of wind loads on the vehicle is fairly straightforward. As the vehicle 
passes through the wind, the upper portion of the vehicle feels a higher wind force 
than the lower portion. This causes the vehicle to rotate in the direction of the wind. 
The vehicle attitude control system reacts to right the vehicle by gimbaling the en- 
gines so as to counter the rotation caused by the wind force. This action introduces 
a vehicle bending moment which, if high enough, could result in a vehicle structural 
failure. 

In the past, wind data obtained from the balloon soundings were used in a man- 
ual calculation procedure to estimate if it was safe to launch. The process was rudi- 
mentary and time consuming. Wind data had to be taken 5 to 6 hours before launch 
to allow time for this manual operation. Since the winds can change significantly in 
a matter of an hour or so, this process was not entirely satisfactory. 

Currently, this computation process has been automated and turnaround time 
greatly reduced. Now wind data obtained as little as 30 minutes prior to launch can 
be processed by the computer. A simulated flight of the vehicle through the winds is 
then made on the computer. Accurate estimates of the bending loads imposed on the 
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vehicle by the wind are thus obtained. In this way a higher confidence of launch 
safety is gained. It is also possible to have the computer determine a booster steer- 
ing program that will minimize wind loads during the launch. 

An example where online data processing as part of an automated checkout sys- 
tem could have avoided a costly delay, and perhaps in another situation a flight fail- 
ure, is shown by an incident that occurred during checkout of the last Atlas -Centaur 
vehicle, which was launched in August 1968. 

A complete systems test was conducted as part of the standard manual vehicle 
checkout procedure on the Atlas -Centaur. This test is a simulated flight sequence 
test where all events that will occur during actual flight are exercised with the ex- 
ception of engine ignition. The vehicle is in the flight configuration with flight um- 
bilicals connected and stage disconnects mated. As an example of the type of actions 
that occur during this test, the interstage electrical plug actuator is considered. 

For the test and for actual flight, the interstage electrical plug is mounted inside the 
actuator. At the time of stage separation, a signal is sent to the actuator which fires 
a powder charge that unlatches the springs in the actuator and pulls the plug apart. 
During the test, the telemetry data from the sensors on the vehicle are recorded on 
strip chart recorders, After the test, the traces are sent to a data analysis group 
for evaluation. Although visual monitoring of these recorders is possible in real 
time, the number of channels of data being recorded makes it difficult. 

This particular test was completed at 6:00 a. m. , and the data were immediately 
delivered to the analysis people. A comparison of each channel of data with a refer- 
ence trace was made. At 3:00 p. m. , an anomaly was noted in a trace of the yaw 
gyro responses, The problem was first thought to be a defective gyro, and the launch 
checkout crew was immediately notified of the anomaly. As already mentioned, the 
vehicle was undergoing a flight sequence test when the anomaly occurred. To rerun 
the test it was necessary to remate all the connections that had been broken as part 
of the test. This included the interstage electrical plug under consideration. After 
this was done, the test was rerun; however, this time no anomaly occurred. The 
temptation to ignore such an intermittent problem has been found to be wishful think- 
ing. Consequently, the flight was canceled until whatever caused the anomaly could 

be found. The following week was spent trying to locate the cause of the trouble. 

%■ * 

After little success, laborious continuity checks were performed following many con- 
nects and disconnects of the various connectors in the faulty circuit. Finally, a pin 
in the flight interstage connector was found to be intermittently bad. 

If automated checkout equipment had been in operation for this test, the anomaly 
would have been noted at the time of the failure and troubleshooting would have 
started before a change in configuration occurred. Then, the testing of only a small 
portion of the vehicle gyro circuits would have isolated the trouble. Although auto- 



mated checkout has not been applied to a launch vehicle in the Atlas -Centaur class, 
incidents such as this have prompted its serious consideration. 

One of the valuable byproducts of computer - controlled testing came about unex- 
pectedly. Since testing procedure is relegated to a computer which relies strictly on 
input instructions, test engineers were required to describe in detail the test to be 
performed, the test stimuli, test limits, and possible failure modes and recovery 
routes prior to the start of testing. The result has been a reduction in hardware loss 
due to out-of-limit testing, poorly run tests, improper recovery action during a mal- 
function, etc. 

Once the prelaunch phase is completed, a second and much shorter phase of the 
launch preparations begins. This is the launch countdown or perhaps better called 
the startup phase; for the Saturn V, this period begins 187 seconds prior to liftoff. 
The launch countdown differs from the pre launch checkout in that the count is more a 
series of specific events programmed to occur during the count. 

Figure 5-13 shows the launch director's countdown panel. Once the launch di- 
rector initiates the count by pressing the start button, all events that occur are auto- 
matically controlled by the computer. The countdown starts with vent valve closure 
and tank pressurization, continues through power transfer and guidance alert, and 
ends with engine ignition and release of the vehicle. Throughout the count sequence, 
a constant comparison of what is occurring to what should occur is made by the com- 
puter. Any out-of -tolerance data results in a preprogrammed halt by the computer 
followed by a recycle to the beginning of the countdown. The many test points in the 
count allow rapid isolation of any malfunction. Recycle can occur at any time - even 
milliseconds prior to release of the vehicle. This is a completely automated portion 
of vehicle checkout and would be analogous to a startup of a boiler-turbine-generator 
unit. 

The postlaunch phase starts once the booster has delivered its payload to its des- 
tination. After all the television coverage that was given to the Mercury and Gemini 
flights, most people are familiar with the checkout procedures followed in an orbiting 
manned spacecraft. This phase of the operation is similar to keeping tabs on a run- 
ning system, and the main concern is watching for malfunctions. For unmanned pay- 
loads, the procedures are truly automated and operation times run into months and 
even years. Even for our manned spacecraft, much of the operation is automated. 

In summary then, it can be concluded from the foregoing that automatic checkout 
of launch vehicles and spacecraft is not only desirable but essential as systems be- 
come larger and more complex. It not only reduces the time required for checkout 
but also increases reliability. Since these are two important factors, the automatic 
checkout of complex systems such as launch vehicles will continue to become more 
prevalent in the future. 
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TABLE 5-1. - SATURN V MEASUREMENTS SUMMARY 


System 

First 

stage 

Second 

stage 

Third 

stage 

Guidance 

and 

control 

Acceleration 

3 

11 

0 

4 

Acoustics 

4 

5 

0 

1 

Temperature 

252 

325 

120 

58 

Pressure 

234 

198 

88 

13 

Vibration 

76 

61 

0 

28 

Flow 

35 

10 

4 

11 

Position 

1 

44 

8 

21 

Guidance and control 

0 

0 

0 

69 

Radiofrequency and telemetering 

0 

0 

0 

26 

Signals 

133 

223 1 

74 

112 

Liquid level 

22 

6 

7 

0 

Volts, amperes, and frequency 

11 

65 

38 

42 

Miscellaneous 

31 

14 

12 

0 

Angular velocity 

3 

3 

0 

33 

Strain 

71 

27 | 

0 

32 

Total 

876 

992 | 

351 

450 
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Figure 5-1. - Atlas booster and Mercury capsule. 



Figure 5-2. - Saturn V vehicle and Apollo spacecraft. 
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Figure 5-3. - Atlas launch complex. 




Figure 5-5. - Automated checkout equipment system. 



Figure 5-6. - Cape Kennedy control room. 
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Figure 5-7. - Saturn V hydrogen loading system. 


Figure 5-8. - Specific event readout panel. 
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Figure 5-9. - Atlas flight programmer. 



b.20.3 ...AT FIRST STAGE VEHICLE POWER PANEL. E8SG 


b.20.3 ... AT FIRST STAGE VEHICLE POWER PANEL. E8b0 

*****CAUTI0N***** ESfaO 

DC POvER MUST BE APPLIED PRIOR TO AC E870 

POWER. E880 

A. POSITION DC POWER SWITCH ON. E890 

1* IOC POWER ON) INDICATOR ILLUM GREEN. E900 

2. (UC POWER OFF) INDICATOR EXTINGUISHED. E910 


CENTAUR JEST PROCEDURE 
AY66-053A-009-130 
22 SEP 67 019 


7. 

7.1 

7.1.1 


POST TEST OPERATIONS 
SHUTOOWN 

/// PERFORM FIRST STAGE AIRBORNE POWER SHUTDOWN AS FOLLOWS? 

... A. TURN OFF GYRO POWER TEST SOX LOCATED IN THE BOOSTER 
POD, RECORD TIME OF DA? — — — , 

♦♦♦♦♦CAUTION***** 

AC POWER MUST BE REMOVED PRIOR TO OC POWER. 

... B. TURN PROGRAMMER POWER TEST BOX SWITCH TO «»0FF** 

(LOCATED IN BOOSTER POD) RECORD TIME OF DAY— . 

... C. PLACE 400 CYCLE POWER SWITCH <S-4> TO THE OFF 
POSITION (2072). 
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Figure 5-10. - Sections of checkout sheet. 
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Figure 5-11. - 1 
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Figure 5-12. - V." 
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